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ESTIMATION  AND  OPTIMAL  CONTROL  FOR 
CONSTRAINED  MARKOV  CHAINS 

by 

D.-J.  Ma1,  A.  M.  Makowski1  and  A.  Shwartz2 
University  of  Maryland  and  Technion 

ABSTRACT 

The  (optimal)  design  of  many  engineering  systems  can  be  adequately  recast  as  a  Markov 
decision  process,  where  requirements  on  system  performance  are  captured  in  the  form  of 
constraints.  In  this  paper,  various  optimality  results  for  constrained  Markov  decision  processes 
are  briefly  reviewed;  the  corresponding  implementation  issues  are  discussed  and  shown  to 
lead  to  several  problems  of  parameter  estimation.  Simple  situations  where  such  constrained 
problems  naturally  arise,  are  presented  in  the  context  of  queueing  systems,  in  order  to  illustrate 
various  points  of  the  theory.  In  each  case,  the  structure  of  the  optimal  policy  is  exhibited. 

I.  INTRODUCTION 

Controlled  queueing  systems  constitute  a  natural  class  of  models  for  a  variety  of  engineer¬ 
ing  applications,  such  as  the  ones  arising  in  computer  communication  networks  and  in  manu¬ 
facturing  environments  [9,10,23].  The  theory  of  Markov  decision  processes  (MDP’s)  provides 
one  of  the  main  tools  to  analyze  performance  optimality  for  many  such  stochastic  systems.  In 
recent  years,  these  methods  were  successfully  used  on  several  simple  queueing  systems,  to  ob¬ 
tain  an  explicit  form  for  the  optimal  control  strategy  [8,12,25,31].  The  model  considered  here 
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is  one  operating  in  discrete  time ;  this  reflects  the  increasing  use  of  digital  implementation  in 
many  of  today’s  systems;  moreover,  under  standard  assumptions  the  analysis  of  some  classes 
of  continuous-time  MDP’s  reduces  to  the  analysis  of  an  embedded  discrete-time  MDP  [16,27]. 
A  large  body  of  knowledge  is  available  in  the  literature  on  this  class  of  problems,  ranging  from 
optimality  conditions  to  structural  properties  of  the  optimal  strategies  to  algorithmic  results. 
The  discussion  is  typically  carried  out  under  the  assumption  that  system  performance  may 
be  captured  through  a  single  cost  criterion  based  on  an  instantaneous  cost  which  depends  on 
the  state,  the  control  action  and  perhaps  time.  Standard  performance  measures  include  the 
finite-horizon  criterion,  as  well  as  the  discounted  and  long-run  average  costs,  both  being  taken 
over  the  infinite  horizon. 

However,  in  many  practical  situations,  conflicting  goals  need  to  be  taken  into  considera¬ 
tion,  a  requirement  that  often  cannot  be  adequately  captured  through  a  single  cost  function. 
Typical  examples  arise  in  computer  communication  applications  where  it  may  be  desirable  to 
maximize  throughput,  while  keeping  delays  small,  or  in  manufacturing  systems  where  resources 
are  allocated  so  as  to  maximize  the  so-called  line  throughput  and  yet  prevent  inventories  to 
build-up. 

Such  trade-off  considerations  require  that  the  standard  formulation  for  MDP’s  be  modified 
so  as  to  accomodate  the  conflicting  objectives.  One  possible  way  to  achieve  this  would  be  to 
define  several  cost  functions,  one  for  each  objective  identified  by  the  designer,  and  to  focus 
attention  on  the  corresponding  multi-objective  optimization  problem.  Here  instead,  conflicting 
requirements  inherent  to  the  (optimal)  design  problem  at  hand  are  incorporated  through 
constraints.  Such  an  approach  was  considered  by  Ross  [26]  in  the  context  of  finite-state 
MDP’s  under  an  objective  criterion  and  a  constraint  function,  each  being  given  in  the  form  of 
a  long-run  average  functional  asociated  with  an  instantaneous  cost,  and  the  goal  is  to  maximize 
the  first  criterion  subject  to  a  bound  on  the  constraint.  As  will  become  apparent  from  the 
discussion  given  in  forthcoming  sections,  constrained  MDP’s  form  a  rich  class  of  stochastic 
optimization  problems  whose  solution  leads  to  a  variety  of  interesting  questions  of  both  a 
theoretical  and  practical  nature. 

This  paper,  which  is  devoted  to  a  brief  survey  of  recent  work  in  this  area,  is  organized 
as  follows:  In  Section  2,  the  single  constraint  optimization  problem  is  precisely  formulated,  a 
solution  technique  via  Lagrangian  arguments  is  then  outlined  and  general  results  on  the  struc- 
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ture  of  optimal  policies  are  summarized.  Extensions  to  more  complex  (i.  e.,  non-Markovian) 
dynamics  as  well  as  to  the  more  difficult  situation  where  several  constraints  are  enforced,  are 
briefly  considered  in  Section  3.  Also  discussed  in  Section  3  are  various  implementation  issues 
which  are  inherent  to  the  very  form  of  the  optimal  policies  identified  in  Section  2,  and  which 
lead  naturally  to  specific  problems  of  combined  estimation  and  control  for  Markov  chains. 
Most  of  the  literature  on  this  topic  [11]  is  concerned  with  indirect  adaptive  control  problems 
[7].  For  constrained  MDP’s,  the  situation  is  somewhat  different  in  that  some  control  param¬ 
eters  may  not  be  available  to  the  decision-maker  in  practice,  even  if  the  model  parameters 
were  known.  This  suggests  viewing  the  design  of  implementable  constrained  optimal  policies 
as  a  direct  adaptive  control  problem  [7].  The  various  ideas  and  proposals  of  Section  3  are 
illustrated  in  two  specific  situations  of  independent  interest,  which  are  discussed  in  Sections 
4  and  5,  respectively.  The  first  situation  centers  around  a  problem  of  resource  sharing,  where 
several  discrete-time  queues  with  geometric  service  requirements  compete  for  the  service  at¬ 
tention  of  a  single  server.  The  second  example  is  essentially  a  problem  of  optimal  flow  control 
for  a  discrete-time  M\M\1  queue. 

2.  MDP’S  WITH  A  SINGLE  CONSTRAINT 
2.1  The  problem  formulation 

In  [26],  Ross  considers  the  following  version  of  the  constrained  optimization  problem  for 
Markov  chains:  Let  {^(n)}^3  denote  a  controlled  Markov  chain,  with  countable  state  space 
S,  compact  metric  action  space  U  and  transition  probabilities  (pXy[u))  assumed  continuous  in 
u.  Following  the  usual  formulation  of  MDP’s  as  given  in  [13],  an  admissible  policy  n  generates 
at  time  n  an  action  U(n)  on  the  basis  of  the  information  H(n)  :=  (X(l),  C/(l) ,  •  •  •  ,X(n  — 
1  ),U{n  —  l),X(n)).  For  a  given  initial  state  distribution  (held  fixed  hereafter),  the  policy  7r 
induces  a  probability  measure  P *  on  the  natural  cr-field  that  equips  the  canonical  sample  space 
0  :=  (5  x  t/)°°,  with  corresponding  expectation  operator  E* .  The  notation  P  is  reserved  for 
the  collection  of  all  admissible  policies.  The  class  of  (possibly  randomized)  Markov  stationary 
policies  is  then  denoted  by  J ,  while  Q  stands  for  the  subclass  of  all  non-randomized  policies 
in  7.  Clearly  Q  C  7  C  P. 

Given  are  two  mappings  r,  c:  S  x  t/  — ►  2R,  which  are  assumed  continuous  in  the  variable 
u  and  which  are  interpreted  as  the  instantaneous  reward  and  cost  functions,  respectively.  For 
every  admissible  policy  n  in  P,  pose 


3 


(2.1) 


J(n)  :=lim„  \e’  £r(X(t),U(t)) 

Tv 

t=  1 

and 

K{tt)  :=  Umn  -E9  J2  <X{t),  U{t)),  (2.2) 

71/ 

t=  1 

and  for  every  V  in  2R,  define 


Pv  :=  {tt  in  P  :  K(n)  <V}.  (2.3) 

Of  interest  here  is  the  constrained  problem  (CPy)  defined  as 

( CPy )  :  maximize  J(n)  over  Py. 

2.2  A  Lagrangian  methodology 

A  Lagrangian  methodology  is  now  described  for  studying  this  constrained  problem;  it 
requires  the  introduction  of  a  family  of  auxiliary  problems:  For  every  7  >  0,  let  the  mapping 
b^-.S  X  U  — *■  TR  be  given  by 

6'Y(x, u)  :=  r[x,u)  —  7 c(x,u)  (2.4) 

for  all  ( x ,  u)  in  S  x  U,  and  define  the  corresponding  Lagrangian  functional  by 

£»  :=  Um„  ijrXV(X(0,  £/(())  (2.5) 

Tv 

t=  1 

for  every  policy  7r  in  P.  As  it  occurs  in  Mathematical  Programming,  the  solution  of  the 
constrained  MDP  (CPy)  is  closely  related  to  various  properties  of  the  unconstrained  MDP 
{LP^)  associated  with  (2.5),  where 

(LP7)  :  maximize  B1  (tt)  over  P . 

To  see  this,  observe  that  for  any  policy  tv  in  P,  the  inequality  B'1(n)  >  J(n)  —  7iif(7r) 
holds,  whereas  if  the  policy  7r  yields  (2.1)  and  (2.2)  as  limits ,  then  £P(7r)  =  J(n)  —  7-ff(7r).  If, 
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in  addition  to  this  property,  a  policy  tt*  meets  the  constraint,  i.  e.,  K(ir*)  =  V ,  and  is  optimal 
for  the  MDP  ( LP ^),  then  necessarily  for  all  ir  in  P , 

B'1( TT*)  =  J( tt*)  -  -/Kin*)  >  B^( tt)  >  J(jt)  -  ^K( tt).  (2.6) 


Since  iT(7r)  <  V  for  any  policy  n  in  Py,  the  inequality  (2.6)  and  the  fact  7  >  0  readily  imply 
that 

J( TT*)  >  J( tt)  +  7[tf(7T*)  -  JT(tt)]  =  J(tt)  +  7(F  -  lir(jr))  >  J(n)  (2.7) 


for  every  policy  ir  in  Py,  whence  the  policy  ir*  solves  the  constrained  optimization  problem 


(civ). 


From  this  discussion,  it  should  be  clear  to  the  reader  in  what  sense  the  Lagrangian  prob¬ 
lems  {(LP-,),')  >  0}  are  useful  for  solving  the  original  constrained  problem.  Indeed,  as  the 
arguments  given  above  indicate,  any  policy  ir*  in  P  which 
(Rl):  yields  the  expressions  J(ir*)  and  K(ir*)  as  limits, 

(R2):  meets  the  constraint  with  K(i r*)  =  V,  and 
(R3):  solves  the  unconstrained  MDP  {LP-,)  for  some  7  >  0, 
necessarily  solves  the  constrained  problem  (CPy).  This  approach  can  be  used  either  directly 
on  specific  problems  mutatis  mutandis,  as  illustrated  in  Sections  4  and  5,  or  it  can  provide  a 
convenient  theoretical  framework  for  establishing  general  results  on  the  existence  and  struc¬ 
tural  form  of  solutions  to  the  constrained  optimization  problem. 


To  simplify  the  discussion,  it  is  convenient  to  assume  that  S  is  finite  and  that  the  controlled 
chain  has  a  single  ergodic  class  under  each  policy  /  in  J.  In  that  case,  under  any  policy  / 
in  7 ,  the  expressions  (2.1),  (2.2)  and  (2.5)  exist  as  limits  and  are  independent  of  the  initial 
condition.  Moreover,  it  is  well  known  that  an  optimal  policy  for  problem  (LP-,)  can  always 
be  selected  to  be  a  pure  strategy  in  Q.  This  follows  by  standard  arguments  based  on  the 
corresponding  Dynamic  Programming  equation  (2.8),  which  states  here  that  for  all  x  in  S,  the 
relation 


B 1  +  h1( x )  =  max[6'1'(x,u)  +  Pxyi^h1  (y)}  (2.8) 

yeS 

holds  for  some  real  constant  B 1  and  some  mapping  h'1:  S  — >  IR.  Their  existence  is  guaranteed 
under  the  assumed  conditions  [13],  with  B"1  identified  with  the  optimal  value  of  problem  (LP-,). 
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It  is  also  well  known  that  if  Q'1  denotes  the  class  of  policies  g'1  in  Q  with  the  property  that 
for  each  x  in  S,  the  action  u  =  g'1(x)  attains  the  maximum  in  the  Dynamic  Programming 
equation  (2.8),  then  any  policy  in  Q'1  is  optimal  for  the  problem  ( LP 7). 

2.3  Optimality  via  randomized  policies 

In  [26],  it  is  shown  under  the  simplifying  assumptions  stated  earlier,  that  whenever  the 
problem  is  “feasible”,  there  exists  a  constrained  optimal  policy  with  a  simple  structure. 
Theorem  [26]:  If  for  some  0  <  7  <  00,  at  least  one  policy  g'1  in  Q'1  has  the  property 
that  K{g"t)  <  V ,  then  there  exists  a  constrained  optimal  policy  f*  in  7  defined  by  a  simple 
randomization  between  two  pure  Markov  stationary  policies  g  and  g  in  Q. 

More  precisely,  there  exist  0  <  7*  <  00,  and  policies  g  and  g  in  Q'1'  such  that 

K(g)  <V<  K(g).  (2.9) 

If  the  randomized  policies  fq,0  <  q  <  1,  are  defined  by 

fq  ■=  qg  +  (1  -  q)g,  (2.10) 

then  /*  =  fg *,  where  the  optimal  bias  q*  is  determined  as  the  solution  to  the  equation 

K(fq)  =  V,  0<q<l.  (2.11) 

The  discussion  of  this  result  can  be  summarized  as  the  search  for  a  policy  in  J  that 
satisfies  the  requirements  (Rl)-(R3)  for  some  value  7*  >  0  of  the  Lagrangian  parameter.  The 
reader  is  referred  to  [26]  for  details. 

2.4  Optimality  via  mixing  policies 

The  existence  of  policies  g  and  g  in  Q1’  with  the  property  (2.9)  can  be  further  exploited 
to  generate  an  alternate  solution  to  the  constrained  MDP  (CPy)  in  the  one-parameter  family 
of  mixing  policies  (7r(p),0  <  p  <  1}.  For  every  p  in  the  unit  interval  [0, 1],  consider  a  two-sided 
coin  biased  so  that  the  events  Head  and  Tail  occur  with  probability  p  and  1  —  p,  respectively. 
To  define  the  mixing  policy  7r(p),  throw  this  biased  coin  exactly  once  at  the  beginning  of 
times,  before  starting  to  operate  the  system.  The  policy  7r(p)  is  defined  as  the  policy  in  P  that 
operates  according  to  g  (resp.  g )  if  the  outcome  is  Tail  (resp.  Head).  It  is  not  difficult  to  see 
that  for  such  a  policy,  the  relations 

J(7r(p))  =  (! -p)J(p) -FpJ(ff)  (2.12a) 
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and 

K(1t(p))  =  (!  ~v)K{g)  +  pK(g)  (2.126) 

hold  true. 

Now,  under  the  non-degeneracy  assumption  K(g )  7^  K(g),  pose  p*  :=  [V  —  K(g)\/[K(jj)  — 
-K’(ff)]  >  0,  (owing  to  (2.9)).  Observe  from  (2.12)  that  the  policy  n (p*)  steers  (2.2)  to  the 
value  V,  yields  (2.1)  and  (2.2)  as  limits  and  that  B7*  (tt(p))  =  (l  —  p)B 7*  (gr)  +  pB7*  ( g )  =  B7* . 
In  other  words,  the  policy  n(p*)  meets  the  requirements  (Rl)-(R3),  and  consequently  solves 
the  constrained  problem  ( CPy ).  It  should  be  noted,  in  contrast  with  the  policy  f*  obtained 
by  randomization  in  Section  2.3,  that  the  evaluation  of  the  optimal  mixing  parameter  p*  is 
immediate,  if  the  values  K[g)  and  K(g)  are  available. 

3.  MORE  ON  CONSTRAINED  OPTIMIZATION 
3.1  Generalizations 

It  is  possible  to  generalize  the  discussion  of  Section  2  in  several  directions: 

Firstly,  it  is  of  interest  to  consider  system  dynamics  where  more  complex  probabilistic 
mechanisms  are  allowed  for  state  transitions  and/or  where  the  state  processes  live  in  more 
general  spaces.  Beutler  and  Ross  [4]  consider  a  version  of  (CPy)  for  general  semi- Markov 
decision  processes  with  finite  state  space  and  compact  action  space.  Nain  and  Ross  [21]  study 
a  specific  constrained  MDP  with  countable  (non-finite)  state  space.  In  both  cases,  similar 
results  on  the  structure  of  the  constrained  optimal  policy  are  reported.  However,  more  work 
seems  needed  as  no  general  theory  is  available  to  date  in  the  case  of  non-finite  state  spaces. 

Secondly,  as  pointed  out  in  the  introduction,  the  main  practical  motivation  for  studying 
constrained  optimization  problems  arises  from  the  desire  to  handle  situations  with  multiple 
(conflicting)  objectives.  The  next  natural  step  would  consist  in  formulating  constrained  MDP’s 
with  multiple  constraints  as  a  nonlinear  programming  problem  in  the  space  of  policies.  More 
precisely,  with  the  notation  of  Section  2,  let  J(tt)  be  the  cost  associated  with  the  policy  n  in 
P ,  and  let  Kx{i r)  denote  the  corresponding  value  of  the  ith  constraint,  1  <  i  <  I.  For  every 
vector  V  =  (Vi,  •  •  • ,  Vjr)  in  IR1 ,  pose 

Pv  :=  {tt  in  P:  JP(jt)  <  V;,  1  <  *  <  /}  (3.1) 

and  define  the  multiple  constraint  problem  ( GPy )  as  in  Section  2.  Very  little  is  known  on  the 
existence  and  structure  of  optimal  solutions  for  problems  of  this  type.  This  probably  could 
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be  traced  back  to  the  fact  that  the  relationship  of  the  corresponding  Lagrangian  problem  to 
the  original  problem  (CPy)  is  far  more  subtle  in  the  multiple  constraint  situation.  In  fact, 
as  of  the  writing  of  this  paper,  it  is  not  clear  on  how  to  obtain  in  general  an  optimal  policy 
through  randomization  and/or  mixing  procedures  similar  to  the  ones  presented  in  Section  2. 
Results  are  available  only  in  particular  instances;  Altman  and  Shwartz  [l]  establish  existence 
of  a  solution  to  a  problem  with  multiple  constraints  for  the  competing  queue  model  considered 
by  Nain  and  Ross  [21]. 

3.2  Implementation  issues 

Even  if  the  policies  g  and  g,  and  the  value  7*  were  readily  available,  the  computation 
of  the  optimal  bias  q*  may  prove  to  be  a  non-trivial  task,  for  it  requires  solving  for  q  in  the 
implicit  equation  K(fq)  —  V  on  the  interval  [0,1],  and  makes  it  necessary  to  evaluate  the 
expression  K(fq)  for  each  0  <  q  <  1.  Both  steps  usually  turn  out  to  be  highly  difficult  ones  in 
many  applications,  and  are  often  possible  only  via  numerical  methods;  this  difficulty  is  clearly 
illustrated  on  the  competing  queue  model  discussed  in  Section  4. 

The  optimal  bias  q*  acts  as  an  internal  parameter;  it  is  available  in  principle  if  the  external 
(or  model)  parameters  (i.  e.,  the  entries  in  the  transition  matrix)  are  known,  but  may  not 
be  easily  available  in  practice  owing  to  computational  difficulties.  Of  course,  this  difficulty  of 
implementation  is  further  compounded  when  some  of  the  external  parameters  are  not  known, 
since  “on  line”  identification  of  the  external  parameters  does  not  provide  a  feasible  means 
to  evaluate  q*.  In  any  case,  this  points  to  adaptive  methods  for  directly  estimating  q *,  now 
treated  as  an  unknown  parameter,  and  this  specifically  for  the  purpose  of  generating  an  optimal 
control;  in  the  terminology  of  adaptive  systems,  this  is  referred  to  as  direct  adaptive  control 
[7].  This  suggests  broadening  the  notion  of  adaptive  control  for  Markov  chains  to  view  it  as 
a  procedure  for  recursively  updating  the  control  to  meet  the  performance  criterion.  Although 
this  is  a  well-known  problem  in  the  general  theory  of  adaptive  systems,  it  seems  to  have  not 
been  studied  much  in  the  context  of  MDP’s,  at  least  to  the  authors’  knowledge. 

The  reader’s  attention  should  be  drawn  to  the  fact  that  direct  adaptive  control  ideas,  with 
q*  regarded  as  an  unknown  parameter,  do  not  always  lead  to  implementable  policies.  This 
was  illustrated  by  Shwartz  and  Makowski  [28,30]  on  the  competing  queue  problem  of  Section 
4. 
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The  implementation  issues  discussed  above  can  be  addressed  in  the  somewhat  more  gen¬ 
eral  context  of  steering  the  cost  (2.2)  to  a  prespecified  value  V :  Given  is  a  parametrized  family 
{fg,0  <  q  <  1}  of  Markov  stationary  policies,  and  assume,  with  the  notation  g  :=  fo  and 
g  :=  fi,  that  K(g)  <  V  <  K(g).  The  problem  is  then  to  find  a  policy  f*  in  the  parametrized 
family  {fq,  0  <  q  <  1}  that  steers  the  cost  (2.2)  to  the  value  V,  i.  e.,  K(f*)  =  V.  If  the  map¬ 
ping  q  — >  K(fq)  is  continuous,  this  can  be  achieved  by  selecting  /*  to  be  fq»,  with  the  bias  q* 
being  determined  as  the  solution  to  the  equation  K(fq )  =  V,  0<q<l.  Although  most  of 
the  ideas  discussed  in  this  paper  apply  mutatis  mutandis  to  this  more  general  situation,  the 
discussion  will  be  carried  out  only  in  the  context  of  constrained  MDP’s  for  sake  of  clarity. 

3.3  A  time-sharing  implementation 


Although  the  optimal  mixing  policy  tt (p*)  of  Section  2.4  has  a  very  simple  structure, 
it  is  not  stationary  and  ergodic.  Indeed,  under  such  a  policy  7r(p*),  the  sample  averages 
corresponding  to  K(ir(p*))  do  not  satisfy  the  constraint  since  on  a  set  of  probability  p*  (resp. 
1  —  p*),  these  limits  will  be  K(g)  (resp.  K(g)).  This  somewhat  unappealing  feature  can  be 
eliminated  through  the  following  timesharing  implementation  of  mixing  policies.  Assume  the 
existence  of  a  privileged  state  to  which  the  system  returns  to  infinitely  often  under  each  one 
of  the  policies  g  and  g,  and  define  a  cycle  as  the  time  T  between  consecutive  visits  to  that 
state.  Denote  the  expectation  of  a  cycle  duration  under  policies  g  and  g  by  E^-[T)  and  Eg(T) 
,  respectively.  For  every  p  in  the  unit  interval  [0, 1],  the  mixing  policy  ?r(p)  has  a  timesharing 
implementation  q:ts(p)  which  is  now  defined:  Let  p  be  the  element  of  [0, 1]  uniquely  defined 
through  the  relation 


„  pE'(T) 

V  (l-p)E’-(T)+pE‘{r) 


(3.2) 


and  consider  two  sequences  of  non 


-negative  integers  {n^f3  and  {rcy}i°  with  the  property  that 


limjn(J)  =  oo 


a  v 

and  hmj—— r  =  p, 

n(J) 


(3.3) 


where  the  notations 

j  J 

n(J)  :=^ny,  n(J)  :=^n3-  and  n(J)  :=  n(J)  +  n(J)  (3.4) 

j= i  i=1 
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are  used  for  every  J  in  IN.  The  discrete-time  axis  is  divided  into  contiguous  control  frames ; 
the  (J  +  l)rst  such  control  frame  starts  upon  completion  of  the  n{J)th  cycle  and  is  made  up 
of  nj+ 1  +  nj+i  cycles.  The  policy  <*ts(p)  is  defined  as  the  policy  in  P  that  during  the  Jth 
frame  operates  policy  g  for  nj  cycles,  and  then  policy  g  for  nj  cycles,  J  =  1, 2,  •  •  •.  Under  the 
condition  (3.3),  well-known  properties  of  first  return  times  for  Markov  chains  readily  imply 
that 


J(q:ts(p)) 


(1  -p)Eg-(T)J(g)+pE<>(T)J(g) 
(1  -  p)Ea-{T)  +  pE9{T) 


J{n(p)) 


(3.5  a) 


and 


lf(a!Ts(p))  = 


(l-p)E<L(T)K(g)+pE°(T)K(g) 

{l-p)E<L(T)+pE*(T) 


K(n(p)) 


(3.5  b) 


where  the  second  equality  is  justified  through  (2.12)  by  the  definition  of  p.  In  fact,  the 
convergence  (3.5)  takes  place  for  the  sample  averages  as  well.  Note  that  if  p  were  rational, 
say  of  the  form  p  =  =qjj^  for  some  integers  n  and  n,  then  the  conditions  (3.3)  would  be 
automatically  satisfied  upon  choosing  n^  =  n  and  ny  =  n  for  all  j  in  IN. 

The  reader  will  readily  see  that  the  time-sharing  implementation  «ts(p*)  of  the  optimal 
mixing  policy  7r(p*),  satisfies  the  requirements  (Rl)-(R3)  and  thus  solves  the  constrained 
problem  ( CPy  )• 

In  Section  4,  this  approach  is  shown  to  be  useful  for  identifying  a  solution  to  a  multiple 
constraint  problem. 

3.4  A  Certainty  Equivalence  implementation 

A  possible  solution  to  the  difficulties  mentioned  in  Section  3.2  would  be  to  estimate 
directly  the  optimal  bias  q*  and  then  use  the  Certainty  Equivalence  Principle  at  each  step. 
More  precisely,  this  suggests  using  a  possibly  recursive  estimation  scheme  that  generates  a 
sequence  of  bias  values  {?(w)}J°  converging  to  q* .  At  step  n,  the  RV  q(n)  constitutes  an 
estimate  of  the  bias  value  q*,  which  is  thus  interpreted  as  the  (conditional)  probability  of 
using  g  (given  available  information  H(n)),  and  it  is  thus  natural  to  select  the  control  action 
U(n)  according  to  /g(n)i  the  adaptive  policy  so  generated  by  the  sequence  {^(w))}^  is  denoted 
by  a. 

There  are  as  many  such  adaptive  schemes  as  there  are  schemes  for  estimating  the  optimal 
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bias  value  q*.  In  each  specific  case,  optimality  of  the  adaptive  policy  a  will  be  concluded  if  it 
can  be  established  that  policy  a 

(Al):.  yields  (2.1)  and  (2.2)  as  limits, 

(A2):  meets  the  constraint,  i.  e.,  K(a)  =  V,  and 
(A3):  yields  the  same  cost  as  /*,  i.  e.,  J{a)  =  J(/*). 

This  is  done  via  a  separate  analysis  and  typically  proceeds  by  showing  that 

lim„|/g(n) (X(n))  -  f*(X(n))\  =  0,  (3.6) 

the  convergence  taking  place  under  Pa  either  almost  surely  or  in  probability,  a  property  which 
readily  follows  from  the  (weak)  consistency  of  the  estimation  scheme.  Under  (3.6)  and  possibly 
additional  structural  model  assumptions,  a  method  of  proof  due  to  Mandl  [19]  can  be  extended 
to  show  that  J(oi)  —  J(f*)  and  K(a )  =  K(f*).  Examples  of  this  approach  are  given  in 
Sections  4  and  5. 

At  this  point,  the  reader  may  wonder  as  to  how  such  an  estimation  scheme  is  selected. 

(i) :  Sometimes,  it  is  feasible  to  compute  the  optimal  bias  q*  as  a  function  q*(0)  of  the 
external  parameters  6.  In  that  case,  the  designer  may  want  to  consider  using  the  Certainty 
Equivalence  Principle  in  conjunction  with  a  parameter  estimation  scheme,  say  based  on  the 
Maximum  Likelihood  Principle.  This  approach  is  illustrated  in  Section  5  on  a  problem  of  flow 
control. 

(ii) :  In  many  applications,  the  function  q  -*■  K(fq)  turns  out  to  be  continuous  and  strictly 
monotone,  say  increasing  for  sake  of  definiteness.  In  that  case,  the  search  for  q*  can  be 
interpreted  as  finding  the  zero  of  the  continuous,  strictly  monotone  function  K(fq)  —  V  and 
this  brings  to  mind  ideas  from  the  theory  of  Stochastic  Approximations  [24].  Here,  this  circle 
of  ideas  suggests  generating  a  sequence  of  bias  values  (g^)}?5  through  the  recursion 

?(n  +  1)  =  [  q(n)  +  an{V  -  c(X(n+  1  ),U(n+  l)))  ]*  n  =  1, 2,  —  (3.7) 

with  g(l)  given  in  [0,1].  Here  [x]q  :=  0  V  (x  A  1)  for  all  x  in  IR  and  the  sequence  of  step  sizes 
{an}i°  satisfies 

oo  oo 

0  <  an  J,  0,  'y  ]  an  =  oo,  y  ^  |  ®n.+i  1*^  oo-  (3-8) 

n— 1  n—1 
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The  corresponding  policy  asA  is  structurally  simple  and  easy  to  implement  on-line;  this  sim¬ 
plicity  of  implementation  is  derived  from  the  fact  that  the  difficult  step  of  directly  solving  for 
q*  is  completely  bypassed. 

4.  OPTIMAL  RESOURCE  ALLOCATION 
4.1  Model 

Consider  the  following  system  of  jK"  -j-  1  infinite-capacity  queues  that  compete  for  the  use 
of  a  single  server:  Time  is  slotted  and  the  service  requirement  of  each  customer  corresponds 
exactly  to  one  time  slot.  At  the  beginning  of  each  time  slot,  the  controller  gives  priority  to 
one  of  the  queues.  If  the  kth  queue  is  given  service  attention  during  that  slot,  then  with 
probability  p,k  the  serviced  customer  (if  any)  completes  service  and  leaves  the  system,  while 
with  probability  1  —  fik,  the  customer  fails  to  complete  service  and  remains  in  the  queue.  The 
arrival  pattern  is  modelled  as  a  renewal  process,  in  that  the  batch  sizes  of  customers  arriving 
into  the  system  in  each  slot  are  independent  and  identically  distributed  from  slot  to  slot.  Under 
these  assumptions,  the  evolution  of  the  system  is  fully  described  by  an  INK+1  -valued  process 
{X(n)}f’,  with  Xk{n),  0  <  k  <  K,  representing  the  number  of  customers  in  the  kth  queue  at 
the  beginning  of  the  slot  [n,  n  +  1). 

The  mean  number  of  customers  arriving  to  the  kth  queue  is  denoted  by  A*.  Define  the 
traffic  intensity  p  :=  )CjfcL0  an<^  assume  henceforth  that  p  <  1;  this  guarantees  system 
stability  [2]. 

For  some  mapping  c:  INK+1  —*  1R,  the  cost  to  be  minimized  is  defined  by 

J(n)  :=  limn-£7r  ^c(X(t))  (4.1) 

71/ 

t=  1 

for  every  policy  n  in  P.  A  special  case  abundantly  treated  in  the  literature  is  the  one  where  c 
is  linear  and  positive,  i.e.  for  all  x  in  INK+1, 

K 

c{x)  =  CkXk ’  (4-2) 

k=0 

with  Ck  >  0, 0  <  k  <  K.  For  this  case,  several  authors  have  discussed  the  problem  of  selecting 
a  service  allocation  strategy  that  minimizes  (4.1)  over  the  class  P  of  all  admissible  service 
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allocation  strategies  [3,5].  They  all  show  the  optimality  of  the  /zc-rule,  i.  e.,  the  fixed  priority 
assignment  policy  that  orders  the  customer  classes  in  increasing  order  of  priority  with  the 
values  /ijtcjtjO  <  k  <  K. 

4.2  Single  constrained  queue 

In  [21],  Nain  and  Ross  considered  the  situation  where  several  types  of  traffic,  e.g.,  voice, 
video  and  data,  compete  for  the  use  of  a  single  synchronous  communication  channel.  They 
formulate  this  situation  as  a  system  of  K  +  1  discrete-time  queues  that  compete  for  the 
attention  of  a  single  server,  and  solve  for  the  service  allocation  strategy  that  minimizes  the 
long-run  average  of  a  linear  expression  in  the  queue  sizes  of  the  K  customer  classes  {l,  •  •  • ,  K} 
under  the  constraint  that  the  long-run  average  queue  size  of  the  remaining  customer  class  0 
does  not  exceed  a  certain  value  V.  Thus  for  any  policy  t r  in  P,  define  J( n)  by  (4.1)  with  c 
given  by  (4.2)  where  cq  =  0,  and  pose 


K( tt)  :=  limn- E^X0(t).  (4.3) 

TV 

Nain  and  Ross  [21]  extend  some  of  the  optimality  results  from  [3,5]  to  show  that  if  the 
constraint  can  be  met  in  a  non-trivial  fashion,  then  an  optimal  policy  with  a  very  simple 
structure  can  be  identified. 

Theorem  [21]:  If  the  problem  is  feasible,  there  exists  a  constrained  optimal  policy  f*  in  7 
which  randomizes  between  two  work- conserving  static  priority  assignment  policies. 

This  result  is  derived  through  a  Lagrangian  argument  in  the  following  way:  For  7  >  0, 
let  the  mapping  b'1:  INK+1  — >  JR  be  given  by  67(x)  =  c(x)  +  7x0  and  pose 

B7(tt)  :=  Y,  67(X(t))  (4.4) 

TV 


for  every  policy  tt  in  P.  The  unconstrained  MDP  ( LP. ^)  is  now  defined  as 

[LP^]  :  minimize  B'1( 7r)  over  P , 

and  solving  the  constrained  problem  ( CPy )  reduces  to  finding  a  policy  tt*  in  P  which  meets 
the  requirements  (Rl)-(R3). 
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If  the  constraint  is  not  satisfied  while  giving  highest  priority  to  the  constrained  queue  (i. 
e.,  the  0th  queue),  then  there  is  no  solution.  On  the  other  hand,  if  the  constraint  is  satisfied 
while  giving  lowest  priority  to  the  constrained  queue  and  ordering  the  other  queues  according 
to  the  /ic-rule,  then  this  policy  is  optimal  for  the  constrained  problem  ( CPy ).  The  proof  in  the 
remaining  case  is  available  in  [21],  the  main  idea  being  that  each  one  of  the  problems  (LP^)  is 
solved  by  a  fixed  priority  assignment  policy  which  admits  a  description  as  the  /xc-rule  based 
on  no^i  and  fikCk,  1  <  k  <  K. 

The  dynamics  of  this  problem  were  generalized  by  Nain  and  Ross  to  a  semi-Markov 
decision  process  [22].  Another  generalization,  given  by  Altman  and  Shwartz  [l],  involves  a 
more  general  constraint  (4.3)  associated  with  an  instantaneous  cost  d:  INK+1  — >  1R  which  is 
also  an  arbitrary  linear  function  with  positive  coefficients  (as  in  (4.2)). 

Theorem  [1]:  If  the  problem  is  feasible,  then  there  is  a  constrained  optimal  policy  fq*  which 
randomizes  between  two  work-conserving  static  priority  assignment  policies. 

4.3  Single  constraint  -  An  adaptive  implementation 

Even  with  (4.3),  the  function  q  — >  K(fq)  is  not  easy  to  compute,  in  spite  of  the  linearity  of 
the  instantaneous  cost.  Indeed,  as  pointed  out  by  Nain  and  Ross  [21],  computing  the  quantity 
K(fq)  amounts  to  studying  a  coupled-processor  problem  whose  solution  can  be  obtained  via 
a  reduction  to  a  Riemann- Hilbert  problem  [6]. 

The  stochastic  approximation  algorithm  that  generates  the  sequence  of  bias  estimates 
{g(n)}J°  here  takes  the  special  form 

q(n  +  1)  =  [  q(n)  +  an(V  -  X0(n  +  1))  ]  *  n  =  1, 2,  •  •  •  (4.5) 

with  <y(l)  given  in  [0,1]. 

For  K  —  1,  there  are  only  two  fixed  priority  assignment  policies,  and  therefore  no  a 
priori  knowledge  of  the  various  statistics  is  required  in  order  to  implement  this  algorithm.  As 
such,  the  proposed  policy  is  implementable  and  constitutes  an  adaptive  policy  in  the  restricted 
technical  sense  understood  in  the  literature  on  the  non-Bayesian  adaptive  control  problem  for 
Markov  chains  [11].  However,  for  K  >  1,  cksa  is  implementable  only  if  g  and  g  have  been 
determined. 
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The  basic  results  assume  finite  third  moments  on  the  initial  queue  sizes  and  on  the  statis¬ 
tics  of  the  arrival  pattern.  Under  these  additional  conditions, 

Theorem  [30]:  The  sequence  of  biases  {?(w)}i°  converges  in  probability  ( under  P“SA)  to  the 
optimal  bias  q*. 

The  derivation  of  this  result  makes  combined  use  of  results  by  Kushner  and  Shwartz  [14] 
on  the  weak  convergence  of  Stochastic  Approximations  via  ODE  methods  and  on  moment 
estimates  obtained  for  the  queue  size  process  in  [30].  The  condition  (3.6)  now  holds  and 
implies 

Theorem  [30] :  The  policy  oisa  solves  the  constrained  optimization  problem  ( CPy )  with 
J{<* sa)  =  J{f*)  and  K(aSA)  =  K(f*). 

4.4  Multiple  constraint  -  A  time-sharing  implementation 

In  this  section,  a  special  version  of  the  multiple  constraint  problem  ( CPy )  is  studied.  For 
every  policy  ir  in  P,  pose 

_  i  n 

K1(tt)  :=  lim ^X*(i),  0  <  i  <  K,  (4.6) 

n  t= l 

and  for  every  vector  V  =  (Vi,  •  •  • ,  Vk)  in  1RK ,  consider  the  constrained  problem 

( CPy )  :  minimize  over  Py ,  (4-7) 

with  Py  defined  by  (3.1). 

There  are  exactly  L  :=  (K  4-  l)!  non-idling  policies  in  §  which  act  as  fixed  priority 
assignments,  the  Ith  such  policy  being  denoted  throughout  by  gi,  1  <  /  <  L.  An  element 
P  —  (Pij  •  • '  5  Pl)  in  IRl  lies  in  the  L-dimensional  simplex  whenever  0  <  pi  <  1  for  all  1  <  l  <  L, 
with  i  Pi  —  1-  The  mixing  policy  7r(p)  associated  with  any  element  p  in  the  L-dimensional 
simplex  is  defined  through  a  procedure  that  generalizes  the  one  discussed  in  Section  2.4: 
Consider  an  L-sided  coin  that  yields  the  Ith  side  with  probability  pi,  1  <  l  <  L  and  throw  the 
coin  exactly  once  at  the  beginning  of  times,  before  starting  to  operate  the  system.  The  policy 
7 r(p)  is  the  one  that  operates  according  to  the  fixed  priority  policy  gi  if  the  outcome  of  the 
throw  is  the  Ith  side,  1  <  l  <  L.  For  every  p  in  the  L-dimensional  simplex,  the  relations 

L 

Ki(n(p))  =  J2plKi(gl),  0  <  i  <  K,  (4.8) 

i=i 
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hold  true  and  suggest  the  following  Linear  Program  ( LP ),  where 


L 

Minimize  53 ziK°{gi )  (4.9a) 

i=i 


subject  to  the  constraints 

L 

53  ziKi(9l)  <Vi,  0  <i<  K ,  (4.9  b) 

l=i 

and 

L 

0  <  z\  <  1, 1  <  /  <  L,  and  53  Zl  ~  ^  (4.9c) 

l=i 

The  relationship  between  the  Linear  Program  (LP)  and  the  original  constrained  problem 
(CPy)  is  formalized  in  the  following  result. 

Theorem  [l] :  If  the  Linear  Program  (LP),  has  a  solution,  say  p* ,  then  the  policy  w (p*)  solves 
the  multiple  constraint  problem  (CPy).  Conversely,  if  problem  (CPy)  has  a  solution,  then  it 
can  always  be  implemented  by  a  mixing  policy. 

The  solution  of  problem  (CPy)  via  mixing  policies  has  the  clear  advantage  of  requiring 
only  the  solution  of  the  Linear  Program  (LP);  this  is  to  be  contrasted  with  the  difficult 
queueing  problem  that  is  required  to  find  the  optimal  randomized  policy  [21].  A  dynamic 
or  time-sharing  implementation  can  also  be  provided  in  the  spirit  of  Section  3.3.  Here  the 
empty  state  is  taken  to  be  the  privileged  state  and  a  cycle  thus  coincides  with  a  busy  cycle 
for  the  queueing  system.  For  sake  of  brevity,  the  discussion  is  restricted  to  the  case  where 
the  element  p  in  the  L-dimensional  simplex  has  all  its  components  rational  with  Pi  —  ^ p 

1  <  /  <  L,  for  some  n  =  (»i,  •  •  ■  ,ul)  in  INL,  where  |n.|  =  Y,i  ni-  Define  the  policy  7r(n.)  in 
P  as  the  one  that  operates  the  fixed  priority  assignment  gi  over  ni  cycles.  With  the  help  of 
results  on  busy  periods  [2,29],  it  is  a  simple  exercise  to  show,  in  analogy  with  (3.5),  that  the 
relations 

L 

^(aTS(P))  =  5]  0  <i<K,  (4.10) 

l=i  |re| 

hold  true.  The  more  general  situation  can  be  treated  exactly  as  in  Section  3.3. 
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5.  OPTIMAL  FLOW  CONTROL 


5.1  Model  and  constrained  problems 

Consider  the  following  flow  control  model  for  discrete-time  M\M\1  queueing  systems 
[17,18]:  At  the  beginning  of  each  time  slot,  the  controller  decides  either  to  admit  or  reject 
the  arrivals  during  that  slot.  An  admitted  customer  joins  the  queue  while  a  rejected  customer 
is  immediately  lost.  A  customer  (if  any)  that  completes  service  in  a  slot  leaves  the  system 
at  the  end  of  that  slot  with  probability  p,  and  fails  to  complete  service  in  that  slot  with 
probability  1  —  fi,  in  which  case  it  remains  at  the  head  of  the  line  to  await  service  in  the  next 
slot.  The  arrival  pattern  is  modelled  as  a  Bernoulli  sequence  with  parameter  A,  independent  of 
the  service  process  which  is  modelled  as  another  Bernoulli  sequence  with  parameter  p.  Under 
these  assumptions,  the  evolution  of  the  system  is  fully  described  by  an  iAT-valued  process 
(X(n.)}i°,  with  X(n)  representing  the  number  of  customers  in  the  queue  at  the  beginning  of 
the  slot  [n,n  +  1). 

The  problem  considered  here  is  formulated  as  the  search  for  a  policy  that  maximizes  the 
throughput  under  the  constraint  that  the  long-run  average  queue  size  does  not  exceed  a  certain 
value  V,  where  the  throughput  and  the  average  queue  size  are  expressed  as 

J(tt)  :=  E  (X(t)  *  0)  (5.1) 

TV 

t=  1 


and 

K{n):=toin±E*J2x{t),  (5.2) 

n  t= l 

respectively,  for  every  admissible  policy  it  on  P. 

A  threshold  policy  is  a  Markov  stationary  policy  in  J,  with  a  simple  structure  determined 
by  two  parameters  L  and  q  in  IN  and  [0, 1],  respectively,  whence  a  threshold  policy  is  denoted 
hereafter  by  ( L ,  q).  According  to  the  threshold  policy  {L,  q),  an  incoming  customer  is  admitted 
or  rejected  wether  the  queue  size  is  <  L  or  >  L;  if  the  queue  size  is  exactly  L,  a  biased  coin 
with  bias  q  is  flipped  and  the  outcome  then  determines  whether  or  not  the  incoming  customer 
can  access  to  the  queue.  The  adopted  convention  interprets  q  as  the  (conditional)  probability 
of  accepting  an  incoming  customer. 
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The  policy  that  admits  every  single  customer  is  denoted  by  (oo,  1).  If  K((oo,  1))  <  V, 
then  the  problem  ( CPy )  corresponding  to  (5.l)-(5.2)  is  trivially  solved  by  (oo,  1).  On  the  other 
hand,  if  K(( oo,  1))  >  V,  then  the  problem  has  a  non-trivial  solution,  which  can  be  shown  to 
be  of  threshold  type.  This  result  is  derived  again  through  a  Lagrangian  argument:  For  7  >  0, 
let  the  mapping  b1:  IN  — >  IR  be  given  by  6'7( x )  =  fil{x  ^  0)  —  7a:  and  define  for  every  policy 
ir  in  P ,  B '7(7r)  as  in  (2.5).  The  unconstrained  MDP  [LP. y)  is  now  defined  as  Section  2,  and  as 
in  previous  instances  where  this  technique  was  used,  solving  the  constrained  problem  ( CPy ) 
reduces  to  finding  a  policy  tt*  in  P  which  meets  the  requirements  (Rl)-(R3). 

The  unconstrained  Lagrangian  problem  (LP~,)  can  be  solved  through  a  tedious  Dynamic 
Programming  argument  that  shows  concavity  of  the  corresponding  value  function. 

Theorem  [17]:  For  every  7  >  0,  there  exists  a  threshold  policy  [L^,q^)  that  solves  the 
unconstrained  Lagrangian  problem  ( LP ^).  Moreover,  for  each  threshold  value  L  in  IN,  there 
always  exists  *y( L )  >  0  so  that  any  threshold  policy  ( L,q ),  with  q  arbitrary  in  [0,  l],  solves  the 
Lagrangian  problem  ( LP 

Threshold  policies  are  thus  unconstrained  optimal.  For  any  threshold  policy  ( L,q ),  the 
quantities  (5.1)  and  (5.2)  exist  as  limits.  Moreover,  K[[L,q))  increases  as  L  increases,  whereas 
for  fixed  L  in  IN,  the  mapping  q  — >  K((L,q))  is  continuous  and  strictly  monotone  increasing 
on  [0,1].  Since  K[( 00, 1))  >  V,  there  exists  L*  in  IN  such  that  K[[L*, 0))  <  V  <  K[[L*,  1)), 
and  the  continuity  and  the  strict  monotonicity  of  q  — ►  K((L*,q))  then  imply  the  existence  of 
q*  such  that  K((L*,q*))=V. 

Theorem  [17]:  If  the  constrained  problem  [CPy)  has  a  non-trivial  solution,  it  can  be  taken 
to  be  a  threshold  policy  f*  =  ( L*,q *). 

The  quantities  L*  and  q*  are  determined  by  the  arrival  and  service  rates  A  and  p,,  and  by 
the  constraint  value  V ;  they  can  be  computed  as  the  unique  solution  to  the  equation 

K[[L,q))  =  V,  L  =  0, 1,  •  •  •  and  0  <  q  <  1.  (5.3) 

This  result  represents  the  discrete- time  analog  of  results  obtained  by  Lazar  [15].  In 
contrast  with  the  competing  queue  problem,  a  closed-form  expression  is  available  here  for  the 
quantity  K((L,q ))  for  all  L  in  IN  and  q  in  [0,1].  However,  as  in  Section  4,  it  is  still  reasonable 
to  seek  on-line  implementations  of  such  a  threshold  policy. 
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5.2  A  Stochastic  Approximation  implementation 

As  in  Section  3.4,  the  two  policies  g  =  (L*,  0)  and  g  =  ( L *,  l)  are  assumed  available,  which 
presumes  only  knowledge  of  L* .  The  following  stochastic  approximation  algorithm  generates 
the  sequence  of  bias  estimates  {?(«•)  }x°  through  the  recursion 

q(n+  1)  =  [  q(n)  +  an(V  -  X(n+  l))  ]*  n  -  1,2,- ••  (5.4) 

with  <7(1)  given  in  [0,1],  where  in  addition  to  the  conditions  (3.8),  the  step  sizes  {an}j°  satisfy 
ai  <  00.  The  RV  q{n)  constitutes  an  estimate  of  the  bias  value  q*  and  is  interpreted 
as  the  conditional  probability  of  using  g ,  or  equivalently,  of  giving  admission  to  a  potential 
customer  during  the  slot  [n,n+ 1)  when  the  queue  size  X(n)  is  equal  to  L*.  With  this  scheme, 
the  control  action  to  be  implemented  is  simply  generated  according  to  the  optimal  threshold 
policy  f*  when  the  queue  size  is  not  equal  to  L* . 

The  key  result  is  obtained  under  a  fourth  moment  assumption  on  the  initial  queue  size,  and 
is  proved  using  ideas  proposed  by  Metivier  and  Priouret  [20]  on  the  almost  sure  convergence 
of  Stochastic  Approximation  algorithms. 

Theorem  [18]:  The  sequence  of  biases  {<?(«.)  }J°  converges  almost  surely  ( under  P01  SA)  to  the 
optimal  bias  q* . 

The  condition  (3.6),  which  is  now  seen  to  hold,  can  then  be  used  to  show  that 
Theorem  [18]:  The  policy  «sa  solves  the  constrained  optimization  problem  ( CPy )  with 
J{<* sa)  -  J{f*)  and  K{aSA)  =  K(f*). 

5.3  The  time-sharing  implementation 

Since  the  quantity  K((L,q))  is  computable  for  all  values  of  its  arguments,  the  values  L* , 
j K(g)  and  K(g)  are  thus  readily  available.  The  optimal  mixing  parameter  p*  can  then  be 
immediately  evaluated,  and  the  optimal  mixing  policy  n(p*)  considered  in  Section  2.4  is  thus 
easily  implementable.  Here,  the  threshold  nature  of  the  policies  g  and  g  suggests  level  L*  as 
a  privileged  state,  and  a  cycle  is  thus  defined  as  the  time  duration  between  consecutive  visits 
to  level  L*.  The  time-sharing  implementation  q:ts(p*)  corresponding  to  tt (p*)  is  defined  as 
in  Section  3.3  and  provides  an  easy  way  to  implement  an  optimal  constrained  policy.  The 
discussion  is  similar  to  the  one  of  Section  3.3  and  will  be  omitted. 
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5.4  An  indirect  adaptive  control  implementation 

All  previous  implementation  schemes  require  the  knowledge  of  the  optimal  threshold  L* . 
In  case  the  external  parameters,  A  and  (i,  are  unknown,  the  parameter  L*  is  certainly  not 
available  and  all  previously  considered  policies  are  thus  not  implementable  in  their  given  form. 

In  such  cases,  it  is  natural  to  consider  a  scheme  that  uses  Certainty  Equivalence  principle  in 
conjunction  with  maximum  likelihood  estimators.  The  sequence  of  estimates  {(A(»),/i(n))}J° 
of  the  true  parameter  (A,  fi)  is  generated  based  on  all  the  past  available  information  by  invoking 
the  principle  of  maximum  likelihood,  and  the  sequence  {(L(n),  ^(n))}^3  is  then  determined  by 
the  estimates  A (n)  and  n{n)  by  solving  the  equation  (5.3).  In  case  there  is  no  solution  to  the 
equation  (5.3)  for  a  pair  (A(ra),  simply  set  L{n)  =  oo  and  q(n)  =  1.  The  control  action 

to  be  implemented  in  the  slot  [n,  n  +  l)  is  then  generated  according  to  (L(n),  g(n)),  and  the 
corresponding  adaptive  policy  is  denoted  by  «ml- 

The  estimation  procedure  relies  on  a  information  pattern  I(n),  richer  than  H(n)  and 
given  by 

I{n)  =  {X(l),E/(t),A(t),B(t),l  <  t  <  n)  n  =  1,2,.  ••  (5.5) 

where  the  {0,  l}-valued  RV’s  U(n),A(n)  and  B{n )  represent  the  control  action  implemented 
in  the  slot  [»,  n  +  1),  the  arrival  during  that  slot  and  service  completion  at  the  end  of  that 
slot,  respectively. 

Using  the  Strong  Law  of  Large  Numbers  and  results  from  the  theory  of  Large  Deviations, 
the  appropriate  version  of  condition  (3.6)  is  shown  to  again  take  place  [18],  the  rate  of  con¬ 
vergence  being  exponentially  fast;  this  last  fact  is  crucial  to  the  basic  result,  which  is  obtained 
under  a  fourth  moment  assumption  on  the  initial  queue  size. 

Theorem  [18]:  The  policy  qiml  solves  the  constrained  optimization  problem  ( CPy )  with 
J(a ml)  =  J(f*)  and  K{a ML)  =  K{f). 

It  is  not  clear  at  this  time  on  how  to  design  direct  adaptive  control  schemes  when  the 
optimal  threshold  L*  is  not  available.  A  stochastic  approximation  algorithm  for  generating 
recursively  a  sequence  of  estimates  for  L*  similar  to  the  one  used  for  q*  naturally  comes  to 
mind;  however,  the  corresponding  scheme  fails  to  work  owing  to  its  sensitivity  to  variations  of 
the  integer- valued  threshold  [18]. 
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